Mutation Analysis for the Real World: Effectiveness, Efficiency, and Proper Tool Support

Evaluating testing and debugging techniques is important for practitioners and researchers: developers want to know whether their tests are effective in detecting faults, and researchers want to compare different techniques. Mutation analysis fits this need and evaluates a testing or debugging technique by measuring how well it detects seeded faults (mutants). Mutation analysis has an important advantage over approaches that rely on code coverage: it not only assesses whether a test sufficiently covers the program code but also whether that test’s assertions are effective in revealing faults. There is, however, surprisingly little evidence that mutants are a valid substitute for real faults. Furthermore, mutation analysis is well-established in research but hardly used in practice due to scalability problems and insufficient tool support. This talk will address these challenges and summarize our recent contributions in the area of mutation analysis with a focus on effectiveness, efficiency, and tool support.